Aligning videos is a crucial task in computer graphics and vision, required for a wide range of applications. However, when aligning video frames, many problems may arise. For example, if a user tries to align videos taken from different camera trajectories and/or at different times, the scene content and appearance may vary greatly due to the change in perspective, the moving scene content, and lighting differences, and cause the video frames that are aligned to have arbitrarily different appearances.